What We Need An Installer To Do

The following are things that the RHQ installation script needs to do in order to prepare the RHQ Server. This includes configuring a stock (out of box) AS7. It also includes things outside of anything AS7 related.

1. Run our dbsetup/dbupgrade scripts to prepare the Oracle or Postgres backend database (create our tables, insert startup data, upgrade schema if this is an upgrade of a previous version of RHQ)

2. Create a default management user (e.g. mgmt-users.properties) - this is so the user can later utilize RHQ to monitor RHQ via the rhq-server plugin

3. Set up all socket bindings (allowing a user to define the ports that the server will bind to)

4. Turn off the deployment scanner (we do not support end users deploying their own apps in the deployments/ directory)

   • <deployment-scanner ... scan-enabled="false" .../>

5. Configure the transaction manager's default transaction timeout

   • <coordinator-environment default-timeout="600"/>

6. Add new logger levels so we can, for example, allow the user to easily turn on RHQ debugging without turning on DEBUG level on the entire root category and below.

   • <logger category="org.rhq" use-parent-handlers="true"><level name="INFO"/></logger>

7. Create a self-signed keystore so we can support HTTPS/SSL encryption both via browser and agent clients

   • We create a rhq.keystore and store it in the app server's configuration directory for use by our code and by our Tomcat web connectors

8. Create a https web connector (using the self-signed cert mentioned above)

   • <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" max-connections="200">
        <ssl key-alias="RHQ" password="RHQManagement" certificate-key-file="/opt/jbossas/standalone/configuration/rhq.keystore" protocol="TLS" verify-client="false" ca-certificate-file="/opt/jbossas/standalone/configuration/rhq.truststore" ca-certificate-password="RHQManagement" keystore-type="JKS" truststore-type="JKS"/>
     </connector>

9. Reconfigure the "redirect-port" of the http web connector to point to the new https connector we created, based on the custom socket bindings we set (also mentioned above)

   • <connector name="http" ... redirect-port="7443"/>

10. Create several security domains for use with our datasources, REST and our LDAP integration

    • <security-domain name="RHQDSSecurityDomain" cache-type="default">
         <authentication>
            <login-module code="SecureIdentity" flag="required">
               <module-option name="username" value="rhqadmin"/>
               <module-option name="password" value="1eeb2f255e832171df8592078de921bc"/>
            </login-module>
         </authentication>
      </security-domain>
      <security-domain name="RHQRESTSecurityDomain" cache-type="default">
         <authentication>
            <login-module code="Database" flag="required">
               <module-option name="dsJndiName" value="java:jboss/datasources/RHQDS"/>
               <module-option name="principalsQuery" value="SELECT PASSWORD FROM RHQ_PRINCIPAL WHERE principal=?"/>
               <module-option name="rolesQuery" value="SELECT 'all', 'Roles' FROM RHQ_PRINCIPAL WHERE principal=?"/>
               <module-option name="hashAlgorithm" value="MD5"/>
               <module-option name="hashEncoding" value="base64"/>
            </login-module>
         </authentication>
      </security-domain>
      <security-domain name="RHQUserSecurityDomain" cache-type="default">
         <authentication>
            <login-module code="org.rhq.enterprise.server.core.jaas.JDBCLoginModule" flag="sufficient">
               <module-option name="hashAlgorithm" value="MD5"/>
               <module-option name="hashEncoding" value="base64"/>
            </login-module>
         </authentication>
      </security-domain>

11. Create an Infinispan cache (I think we needed this for our REST stuff, but not sure)

    • <cache-container name="rhq" jndi-name="java:jboss/infinispan/rhq">
         <local-cache name="rhqCache">
            <transaction mode="NONE"/>
            <eviction strategy="LRU" max-entries="50000"/>
            <expiration max-idle="100000"/>
         </local-cache>
      </cache-container>

12. Create several JDBC drivers - one for each DB we support (today, that is Postgres and Oracle)

    • <driver name="postgres" module="org.rhq.postgres">
         <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
      </driver>
      <driver name="oracle" module="org.rhq.oracle">
         <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
      </driver>

13. Create the datasources using the database vendor the user wants (oracle or postgres)

    • The main RHQ datasource is a XA datasource (<xa-datasource jta="true" jndi-name="java:jboss/datasources/RHQDS" pool-name="RHQDS" enabled="true" use-java-context="true">) but because we use Quartz, it needs a no-tx datasource (<datasource ... jta="false" .../>)

14. Create several JMS queues

    • <jms-queue name="AlertConditionQueue">
         <entry name="queue/AlertConditionQueue"/>
         <durable>true</durable>
      </jms-queue>
      <jms-queue name="DriftChangesetQueue">
         <entry name="queue/DriftChangesetQueue"/>
         <durable>true</durable>
      </jms-queue>
      <jms-queue name="DriftFileQueue">
         <entry name="queue/DriftFileQueue"/>
         <durable>true</durable>
      </jms-queue>

15. Define SMTP server endpoint - we would like to support secure settings and SMTP proxies but do not today, we only support host/port customizations

    • <outbound-socket-binding name="mail-smtp">
         <remote-destination host="localhost" port="${rhq.server.email.smtp-port:25}"/>
      </outbound-socket-binding>

16. Disable the out of box AS7 default welcome page

    • <virtual-server name="default-host" enable-welcome-root="false"> ...

17. Add custom extension module - this is the startup extension code - it is what houses the RHQ EAR and is what deploys the ear at startup

    • <extension module="org.rhq.rhq-enterprise-server-startup-subsystem"/>

18. After all of the configuration is done, we then reload the app server so it picks up all the changes we made

19. Add custom startup subsystem - when the installer does this, the EAR is automatically deployed

    • <subsystem xmlns="urn:org.rhq:startup:1.0"/>

We will also need Cassandra integrated too. Installing it, upgrading it, populating it and configuring our RHQ Server to be able to find the cluster so it can use it.

Due to some limitations in AS7 (specifically, some configuration settings do not support ${x} expressions), we need some way to reconfigure the AS7 whenever our properties file is changed. Today, we do this by invoking the installer script with the --reconfig command line option everytime we start the server. If we go away from the concept of a single "rhq-server.properties" (and instead, say, rely on end users to edit standalone-full.xml to configure the RHQ Server), then this problem goes away.

Future Plans for Product Layout on File System

Here's the wiki Brian created that is tracking the proposal for product layouts:

https://community.jboss.org/wiki/LayeredDistributionsAndModulePathOrganization

So, in short, for us to comply with this new proposed way for layered products to build on top of AS7 and be patchable using the new patch mechanism, more work is going to be necessary to adhere to this way of productization. More work in our mvn build infrastructure and probably more work in our installer (but hopefully not much more in the installer, all the work we did was to help avert massive changes required if something like this happened).

Here's the long story:

Up until this move to AS7, we've always included AS 4.2.3 inside of the RHQ server distribution. You unzip, and you get our scripts, and things like java service wrapper, but you also get a directory "jbossas" - which is the AS 4.2.3 installation, pre-configured for our needs (i.e. it wasn't a straight AS 4.2.3 download, it was highly customized).

With our move to AS7, we kept changes to the file layout minimal and consistent with how it was done before. The way we have in master now is we still include inside of the RHQ Server distro "jbossas" but this time it is a unadulterated AS7 as it comes straight from the mvn repo (caveat - that's not 100% true - it is minus some things we remove from it, like documentation, the domain stuff, and things like that - but the standalone server and everything around it (jars, config files, startup script, CLI script, etc) remain intact and untouched by us). In addition, our RHQ Server still has (as it has always) some files external to "jbossas" - specifically bin/ scripts to start and install the server as well as modules/ which contains custom code that we need (this includes the Postgres JDBC driver and the Oracle JDBC driver as well as our startup extension which itself has our huge EAR inside of it).

Now, what is this new proposed way for Jboss products to layer on top of AS/EAP? Well, it is a base AS7 distribution with the layered product "inside" of it. This means, our custom modules don't go OUTSIDE of "jbossas", rather, they now should go INSIDE "jbossas" - specifically, something like "jbossas/modules/internal/layers/rhq". Our script files will go, not in our own "<rhq-install-dir>/bin" directory, but in "jbossas/bin". We probably will still need rhq-installer.sh/.bat (because we need to setup the database at minimum) but rhq-server.sh might be able to go away (we just piggy back on AS7's startup script). This also means we rely on AS7's wrapper functionality (to install itself as a boot service, for example - or on Windows, as a Windows Service). I haven't looked at AS7/EAP's support of Windows services (java service wrapper) but I assume they support it and we should just piggy back on it - somehow.

Everything we have, in other words, should go INSIDE of the AS installation directory ($JBOSS_HOME) - we don't have a separate RHQ_SERVER_HOME anymore (or, at most, it should be identical to JBOSS_HOME). Things are still in flux - I'm not sure how it is envisioned for layered products to install their own custom files that aren't related to AS7 explicitly (like, e.g. our plugins directory, or our etc/ directory, or our alert-scripts directory). I assume everything needs to go under JBOSS_HOME somewhere.

Anyway, as you can see, our filesystem layout needs to change. We need to flip everything on its head - AS7 is no longer inside of us, we are to be inside of AS7. And we should be reusing what we can from AS7 (so, that means, we start the server via "standalone.sh" and we just customize the startup by customizing "standalone.conf" or something to that effect - I think if need be we can still ship with a rhq-server.sh, but it would be nice to simply rely on the AS7 startup mechanism that it comes with).